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ABSTRACT 



This document analyzes the following five large-scale 
studies of education technology: (1) "Meta-Analytic Studies of Findings on. 

Computer-Based Instruction" (J.A. Kulik) employed a statistical technique 
called meta-analysis to aggregate the results of over 500 individual studies 
to draw a single conclusion/ (2) "Report on the Effectiveness of Technology 
in Schools, 1990-1997" (J. Sivin-Kachala) reviewed hundreds of individual 
studies whereby the authors shed light on consistent patterns that emerged 
across studies/ (3) "Evaluating the Apple Classrooms of Tomorrow" (E.L. 
Baker, M. Gearhart, & J.L. Herman) reviewed a partnership between Apple and 
five schools across the nation/ (4) "West Virginia's Basic Skills/Computer 
Education Program: An Analysis of Student Achievement" (D. Mann, et al . ) 
reported the results of West Virginia's 10-year statewide education 
technology initiative/ and (5) "Does It Compute? The Relationship between 
Educational Technology and Student Achievement in Mathematics" (H. 
Wenglinsky) assessed a national sample of fourth- and eighth-grade students 
using newer simulation and higher order thinking technologies. An evaluation 
of two smaller- scale studies that point to the promise that newer 
technologies currently afford is also included: "Computer Support for 
Knowledge -Building Communities" (M. Scardamalia & C. Bereiter) and "Software 
Design as a Learning Environment" (I. Harel & S. Papert) . Conclusions are 
summarized related to impact and effectiveness. (AEF) 
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The Impact of Education Technology 
on Student Achievement" 

What the Most Current Research Has to Say 



Legislators, governors and other policymakers each year make difficult choices among attractive 
educational improvement options. Whether they invest in class size reduction, teacher training, 
early childhood education, or textbooks and tests turns on their estimate of the effectiveness of 
these approaches. 

The purpose of this briefing is to outline what we know about the impact of education technology 
on learning and to identify resources for further study. 

Research on the impact of technology on learning is in its infancy though we are beginning to see 
solid work emerge. In this report we look at some large scale state and national studies as well 
as some innovative smaller studies that provide visions for new uses of technology in learning 
and instruction. 



Research on Education Technology's Impact 

Below we analyze the 5 largest scale studies of education technology to date. These studies were 
selected for their scope, comprehensive samples, and generalizability to local, state, and national 
audiences. We also include an evaluation of two smaller scale studies that point to the promise that 
newer technologies currently afford. 

The first study we analyze employed a statistical technique called meta-analysis to aggregate the 
results of over 500 individual studies to draw a single conclusion. The second reviewed hundreds 
of individual studies whereby the authors shed light on consistent patterns that emerged across stud- 
ies. The third reviewed a partnership between Apple and five schools across the nation. The fourth 
study reported the results of West Virginia's 1 0 year statewide education technology initiative. The 
fifth assessed a national sample of fourth- and eighth-grade students using newer simulation and 
higher order thinking technologies. The sixth and seventh reviewed two smaller scale studies that 
show the promise of newer emerging technologies on student learning. 

The intent of this document is to briefly summarize the positive and negative impact of 
various technology studies on student achievement. More in depth information about several of the 
studies reviewed within this briefing can be found by accessing this report at the Milken Exchange 
Web site ( www.milkenexchanae.org ). 
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2. Sivin-Kachala's Review of the Research 



Jay Sivin-Kachala (1998) reviewed 219 research studies from 1990 to 1997 to assess the effect 
of technology on learning and achievement across all learning domains and all ages of learners. 
From his analysis of these individual studies he reported the following consistent patterns: 

Positive Findings 



o Students in technology rich environments experienced positive effects on achievement in all 
major subject areas. 

o Students in technology rich environments showed increased achievement in preschool through 
higher education for both regular and special needs children. 

o Students' attitudes toward learning and their own self-concept improved consistently when 
computers were used for instruction. 

Inconclusive Findings 



o 



The level of effectiveness of educational technology is influenced by the specific student 
population, the software design, the educator's role, and the level of student access to the technology. 



3. The Apple Classrooms of Tomorrow (ACOT) 

In their evaluation of the Apple Classrooms of Tomorrow, Baker, Gearhart, and Herman (1994) 
assessed the impact of interactive technologies on teaching and learning in five school sites across 
the nation (e.g., California, Tennessee, Minnesota, and Ohio). The goals of ACOT were to encour- 
age instructional innovation, and to emphasize to teachers the potential of computers to support 
student initiative, long-term projects, access to multiple resources, and cooperative learning. Over 
the course of the five year initiative, comparisons were made of a) ACOT students' basic skills 
performance to nationally reported norms, b) ACOT students' progress and achievement over time, 
and c) ACOT teachers' teaching practices. 

Positive Findings 



° The ACOT experience appeared to result in new learning experiences requiring higher level 
reasoning and problem solving, although the authors claim this finding was not conclusive. 

o ACOT did have a positive impact on student attitudes and did have an impact on changing 
teacher teaching practices toward more cooperative group work and less teacher 
stand-up lecturing. 



Negative Findings 
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Dde Mann's ^(’^1^99) stud)^,of the stq^e of^West Virginia's Basic{Skills/Corhputer Education (BS/CE) 
program analyzed^ representative'^ sample of 950 fifth-gr^de^ students' achievement Vrom 1 8f 
^elementar;y schools across^e state^These fifth-grade students iVad been participating in the West 
/ Virginia/BS/CE program since- 199 1-92. Data was -also collected from 290''teachers to sfiow tfie 



influence that West ^Virginia's Integrated^ Learning System technology ha^djonMtudentja^ 

^^he Integrated Learning Systery' technology focused its^teaching“On-spelling]j^voc'abd 
and^'mathematics./Several variables^ were collected an^ analyzed i.e., inl^ensitiyity of-Basic Skills/ 
Comf^ter Education (BS/CE), (student prior achievem^ent and sociodemography, teacher traimr^ig, 
teacherN^nd student attitudes toward BS/CE. The findings for West Virginia's statewide initiative 
were as follows: 



/ Positive Findings 








The more students participated in\BS/GE 7 ~the_mbre their test scores rose on_.thej5tghford'9: 
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Consistent student access' to the technology, positive attitudes-tdwards'lhe technology |by both 
tochers and students), ^and teacher\traming jin the technology led to the greafes|Jstudent\ 
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1 i:j : — ^L; 1 computers in the West Virginia study. 
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Lewis Solmo^n's (T9-99) costj benefit analysis of the West Virginia Basic Skills/Compute'r-Education 
prograrn^compared tKe^cost of BS/CE and its achievement gains to the cost of other reforiiji 
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BS/CE was rriorevC^t effective in improving student ocHiievement thafT(l) class size reduction / 
from 35 to 20 stud,ents>42). increasing instructional time, and (3) cross age tutoring programs.y 
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5. Harold Wenglinsky's National Study of 
Technology's Impact on Mathematics Achievement 

Harold Wenglinsky (1998) assessed the effects of simulation and higher order thinking technologies 
on a national sample of 6,227 fourth graders and 7,146 eighth graders mathematics achievement 
on the National Assessment of Educational Progress. Wenglinsky controlled for socioeconomic 
status, class size, and teacher characteristics. Thus, all relationships between technology and 
educational outcomes reported represent the value added by technology for comparable groups of 
students with comparable teachers in comparable class sizes. Wenglinsky found: 



Positive Findings 



o Eighth-grade students who used simulation and higher order thinking software showed gains 
in math scores of up to 15 weeks above grade level as measured by NAEP. 

o Eighth-grade students whose teachers received professional development on computers 
showed gains in math scores of up to 13 weeks above grade level. 



o 



Higher order uses of computers and professional development were positively related to 
students' academic achievement in mathematics for both fourth- and eighth-grade students. 









SOURCE: Educational Testing Service, "Does it compute?" an analysis of 1 996 National Assessment of Educational Progress. 
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Expectations for Depth ofNdentifying Conceptual 
Knowledge Growth Expectation^ A^Diffic^lties 



Source: Scardamalia, M., & Bereiter, C.: CSILE Learning Gains^(1996) 
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7. The Learning and Epistemology Group at MIT 



Seymour Papert, Mitchel Resnick, Yasmin Kafai, and Idit Harel have employed learning by design 
principles to educational technology by having students become creators and designers of educa- 
tional software. These researchers use the computer as the machine to be acted upon and students 
as the actors. Thus, children learn through design activities by programming computers to create 
applications that other children use and learn from. 

Research by Idit Harel (1988; 1991) introduced Logo programming to design software to teach 
fractions to younger students. Students had to structure their computer program, maintain connec- 
tions between content and functionality, and design the user interface and activities. In addition, 
students needed to consider different ideas about how to teach fractions to younger students. 
Harel's research demonstrated that: 

o Students who designed fraction software for other students using Logo learned fractions better 
than students taught fractions using conventional methods. 

o Students who used Logo to design software learned Logo better than students who received 
Logo programming instruction only. 



Conclusion on Impact and Effectiveness 

These studies show that in over 700 empirical research studies, in the study of the entire state of 
West Virginia, in a national sample of fourth- and eighth-grade students, and in an analysis of newer 
educational technologies that students with access to 

(a) computer assisted instruction, or 

(b) integrated learning systems technology, or 

(c) simulations and software that teaches higher order thinking, or 

(d) collaborative networked technologies, or 

(e) design and programming technologies, 

show positive gains in achievement on researcher constructed tests, standardized tests, and 
national tests. 
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There is, however, evidence^ in^^pme' of^the^^ sTudies that learning te^chnologyvr^ less. effective _^r' 






ineffective when the learriing^obiectives are, unclear andrthe fojcus 6Pthe''fe^ is. diffuse..") \^', 

"One of the enduring difficulties about technology dn^.'educatipn," According' to Dr.^M L x / ^ 



Wiske, co-directbr of the -, Educational TechnologyvCehter ' af-the\Harvard^.Grdduqte.-School"^6f 
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To assist! educators arid policymdkers in putting -educatibn first, /the Milken Exchange ser^^^ \ 

as ,d clearinghouse of research /and information on learning technology. To fapijitate^ policy\ 
development and planning, /the Exchange has developed its Seven' D/mens/ons' of Learning. \ 
Technology. By paying attention to^lhe learner, the learning environment, professional competency, , / \ 

/ system capapit^^sCommunity connectic ‘ ' 

/[ be kept in , service to-learhing. O; . 
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